Looking under the hood of OpenShift and uncovering the secret of the Origin repository

If you've worked with OpenShift or installed its free distribution OKD, you've definitely come across the openshift/origin repository. In the era of OpenShift 3 and early 4.x releases, this is where the core of the entire platform lived. Developers cloned a massive portion of the Kubernetes codebase here and built their components on top of it.

But if you go to this repository today, you won't see the old structure there. No familiar controller files, no source code for the hyperkube binary. Where did everything go, and why does Red Hat keep a project with nearly nine thousand stars?

Where did the OpenShift source code go

In the summer of 2020, before the release of OpenShift 4.6, the development team reorganized. The monolithic approach had become an obstacle: synchronizing changes with upstream Kubernetes in one place alongside their own tests had become too complicated.

As a result, the codebase was split:

• All work with the Kubernetes fork and building binaries like hyperkube moved to the openshift/kubernetes repository.
• The openshift/origin repository was turned into a specialized test hub.

Now the main purpose of Origin is to serve as a home for the openshift-tests binary and a set of e2e scenarios that verify cluster compliance with OpenShift and Kubernetes standards.

How end-to-end testing works in openshift-tests

Building tests in the project has nothing to do with running regular go test. Here, a full-fledged openshift-tests binary is compiled, packed inside with hundreds of integration and e2e scenarios.

The OpenShift team has a strict rule for writing e2e tests. Two different tests should not duplicate each other's functionality by more than 10%. Forget about meticulously checking every validation error in the API. The purpose of these tests is to follow a real user journey from start to finish: deploy an application, verify network policies work, ensure routing is correct, and collect metrics.

You can compile the test tool with a single command from the project root:

make

The resulting binary can run both standard Kubernetes conformance tests and narrow-specific checks for Red Hat components.

Environment selectors instead of annotations

In the past, to skip an incompatible test on a specific cluster configuration, engineers would attach annotations directly in the Go code. This created chaos during upgrades.

In modern Origin branches, annotations have been eliminated. Now filtering is controlled by so-called environment selectors (environment selectors). The framework looks at the target cluster parameters before running (such as network provider type or cloud platform) and filters out unsuitable tests on the fly.

The exclusion logic is split into two levels:

• Exceptions for standard Kubernetes tests are located in openshift/kubernetes in the environment_selectors.go and disabled_tests.go files.
• Rules for OpenShift-specific tests live directly in Origin in the pkg/test/extensions directory.

If you're writing your own operator for OpenShift, this scheme makes it easy to understand why a particular upstream test doesn't run in your environment.

Dependency synchronization and Go checksum gotchas

Since origin depends on a fork of openshift/kubernetes, developers have to constantly update Go modules. To avoid doing this manually, a hack/update-kube-vendor.sh script was added to the project.

You can run vendor updates for a specific branch or SHA commit like this:

./hack/update-kube-vendor.sh master

The script can pull changes even from unmerged pull requests. To do this, pass your fork address as the second argument:

./hack/update-kube-vendor.sh my-feature-branch github.com/myname/kubernetes

When working with this script, it's easy to hit a nasty error. Go's checksum proxy (sum.golang.org) sometimes returns 410 Gone if a commit was just created and the checksum database hasn't had time to index it yet.

It looks like this:

go: k8s.io/[email protected] ... 410 Gone
        server response: not found

The solution here is simple — forcibly disable checksum database verification during vendor updates:

GOSUMDB=off hack/update-kube-vendor.sh master

Quickly running external examples

Besides tests, a useful hack/update-external-example.sh script remains in the repository. It downloads up-to-date application manifests and quick starts from third-party ecosystem repositories and places them in the examples folder.

If you need known-to-work examples of Deployment, Route, or StatefulSet for OpenShift, it's worth checking the examples/quickstarts folder — it contains verified configurations.

Who benefits from the Origin repository today

If you just operate an OpenShift cluster, you won't need to dig into Origin code every day. But the project will be a great help in three cases:

• You're writing your own operators or platform extensions and want to run official e2e checks in your CI/CD pipeline.
• You're contributing to OKD development or debugging a custom Kubernetes build for specific hardware.
• You want to see how distributed systems testing architecture in Go is implemented in large-scale commercial projects.

The repository is open under the Apache 2.0 license, and an active community maintains branches for all current platform versions.