CircleCI: Cans and Can'ts

Disclaimer: this article does not cover other CI platforms. Yes, even Jenkins (who uses Jenkins anyway in MMXXIV AD?).

If you're reading this, there is a chance that you know what CircleCI is, and, if you don't, well.
There are moments when you need to build something not very trivial and instead of giving you referentially transparent tooling to achieve that, the platform compels, if not submits, you to employ occult practices to get away with what you want to happen. This article should give you a rough idea on some pitfalls that lack an immediate explanation where it should be explained (CircleCI docs).

First of all, what is the Can #1?

Can #1: Orbs

Technically they are equal to Actions, but seem to be more direct about what they offer and what not really. This is important and I will mention them later.
And then, Can #2?

Can #2: Anchors

This is where YAML quirks collide with the platform.
The first thing you may have been tempted to do is open a fantastic read of CircleCI docs about YAML and be surprised you not only need to know about the existence of anchors but also be tempted to use them. That's where can'ts bonk you the first time.
First of all, you can use list of steps in anchors. That may be obvious, but from docs it isn't.

higherAnchor: &hARef # higher anchor reference
  - *lARef # lower anchor reference
  - *lARef2
  - checkout

It just happened that checkout is its own thing, yet you can use it inside anchor which you then can use literraly in the job spec:

  letCoolStuffHappen:
    <<: *dockerConfigAnchor # you can literally have anchors of docker/executor config
    steps: *hARef

But what if you want to do something interesting, say, use anchors with orbs?
Let's pretend you decided to use amazing and mighty Kubernetes orb - first thing you will most likely think about is:

kubeParamAnchor: &kParams
  kubernetes/create-or-update-resource:
    get-rollout-status: << pipeline.parameters.get-rollout-status >>
    namespace: kekus
    watch-timeout: << pipeline.parameters.watch-timeout >>
    resource-file-path: "deployments/kekus-web.yaml"
    resource-name: "deployment/kekus-web"

So far so good.

higherAnchor: &hARef # higher anchor reference
  - *lARef # lower anchor reference
  - *lARef2
  - checkout
  - *kParams

Let's make it more realistic:

kekusWeb: &kekusWeb # higher anchor reference
  - *lARef # lower anchor reference
  - *lARef2
  - checkout
  - *kParams

And then you need to define a new anchor which is half the same information:

kubeParamAnchorApi: &kParamsApi
  kubernetes/create-or-update-resource:
    get-rollout-status: << pipeline.parameters.get-rollout-status >>
    namespace: kekus
    watch-timeout: << pipeline.parameters.watch-timeout >>
    resource-file-path: "deployments/kekus-api.yaml"
    resource-name: "deployment/kekus-api"

Your initial thought (after panic of course) will be "why not sunder repeating information from specific one", and you'd be right. Except for one thing. How?
Above-mentioned docs give some abstract example. So, you may try out something like:

kubeConst: &kConst
  kubernetes/create-or-update-resource:
    get-rollout-status: << pipeline.parameters.get-rollout-status >>
    namespace: kekus
    watch-timeout: << pipeline.parameters.watch-timeout >>
kubeWeb: &kWeb
  # but then what?

Let's try something and see what we get:

kubeConst: &kConst
  kubernetes/create-or-update-resource:
    get-rollout-status: << pipeline.parameters.get-rollout-status >>
    namespace: kekus
    watch-timeout: << pipeline.parameters.watch-timeout >>
kubeWeb: &kWeb
  <<: *kConst
  resource-file-path: "deployments/kekus-web.yaml"
  resource-name: "deployment/kekus-web"

This will result in schema violation if you try to:

kekusWeb:
  <<: *dockerConfig # this is still valid
  steps: *kWeb # schema violation

You might wonder what's wrong with doing as you're said. Well, everything. And that's the...

Can't #1: Using anchors as you'd think you should

While you can technically infinitely nest anchors, you can't just sunder orb actions in half and get away with it.
Anchors abide principle of 1 anchor - 1 action, and any attempt to break out of this is schema violation (in half of the cases you won't know it until you footgun at runtime):

kekusWeb:
  <<: *dockerConfig # this is still valid
  steps: <<: *kubeWeb # missing required parameters in parent anchor

kekusWeb:
  <<: *dockerConfig # this is still valid
  steps: <<: [*kubeConst, *kubeWeb] # schema violation

Since docs don't explain how to effectively use anchors and orbs, it is time to figure out on our own. What is considered an orb action here?

kubeConst: &kConst
  kubernetes/create-or-update-resource:
    get-rollout-status: << pipeline.parameters.get-rollout-status >>
    namespace: kekus
    watch-timeout: << pipeline.parameters.watch-timeout >>

You would be surprised, but the direct indicator of action (which can require attributes) is kubernetes/create-or-update-resource. So what about redelegating action hierarchy? We can do that, because it is the thing docs are correct about:

kekusWeb:
  <<: *dockerConfig # this is still valid
  steps:
    - kubernetes/create-or-update-resource:

What now? Well, since we announce that we command action to happen earlier than anchor is evaluated, we need to untie parameters from action:

kubeConst: &kConst
  get-rollout-status: << pipeline.parameters.get-rollout-status >>
  namespace: kekus
  watch-timeout: << pipeline.parameters.watch-timeout >>
kubeWeb: &kWeb
  resource-file-path: "deployments/kekus-web.yaml"
  resource-name: "deployment/kekus-web"

Now these anchors have no context of orb action, and we just can:

kekusWeb:
  <<: *dockerConfig # this is still valid
  steps: 
    - kubernetes/create-or-update-resource:
      <<: [*kubeConst, *kubeWeb] # object limit of 1 violation

kekusWeb:
  <<: *dockerConfig # this is still valid
  steps: 
    - kubernetes/create-or-update-resource:
        <<: [*kubeConst, *kubeWeb] # valid indentation

"It just works" - Todd Howard

This way we actually pass the inherited parameters, and, since their combination satisfies orb action requirements, it becomes valid operation.
But what could go wrong after that?

Can't #2: Environment variables

Ever wanted to make some cool environment variables and then pass them around between jobs like aux cable? Well... here you go.

echo 'export KEKUS=$(echo "$KEK")' >> $BASH_ENV

God knows why it's like that, and there's no really a way around this. Funnily enough, attempt to overwrite predefined variables will be silently rejected. Of course you can:(source, god save that poor man's soul)

jobs:
  create-env-var:
    executor: basic
    steps:
      - run: |
          echo "export FOO=BAR" >> $BASH_ENV
      - run: |
          # verify; optional step
          printenv FOO
      - run: |
          cp $BASH_ENV bash.env
      - persist_to_workspace:
          root: .
          paths:
            - bash.env
  load-env-var:
    executor: basic
    steps:
      - attach_workspace:
          at: .
      - run: |
          cat bash.env >> $BASH_ENV
      - run: |
          # verify; this should print 'BAR'
          printenv FOO

And yes, if you decided that hundreds Mb images are not for you and you opted in for alpine, be ready to find out that suddenly $BASH_ENV doesn't work, and you need to resort to even more occultism:

- run:
    command: |
      echo 'export KEK=$KEKUSVAR' >> $BASH_ENV
      echo 'source $BASH_ENV' >> $HOME/.bashrc
- run:
    shell: bash # otherwise it won't work
    # default shell in alpine is sh/ash
    ...

...but since the article is about retaining sanity, it is time to mention the can't which derives right from this.

Can't #3: Multiple workspaces

Ever wanted to persist multiple items and then conveniently retrieve them to their exact location without having to cache the whole thing or something? Well, guess again:

"attach_workspace": {
  "allOf": [
    {
      "$ref": "#/definitions/builtinSteps/documentation/attach_workspace"
    }
  ],
  "type": "object",
  "additionalProperties": false,
  "required": ["at"],
  "properties": {
    "name": {
      "description": "Title of the step to be shown in the CircleCI UI",
      "type": "string"
    },
    "at": {
      "description": "Directory to attach the workspace to",
      "type": "string"
    }
  }
}

While you can persist multiple items at once (and you will need to be consistent about it if you want to see them somewhere later back in one piece), you will need to explicitly distribute them to their respective locations if those don't belong to same location. Named workspaces? Nah, we don't do that here. Bash script to the rescue.
And while you may have thought you got it covered:

persist_to_workspace:
  root: ~/
  paths:
    - $KEKUSVAR.exe

Nope. You are not allowed to interpolate and that's about it.
Speaking of which...

Can't #4: Bash just doing what you want it to

This section kind of derives from #2, I will leave a single example of what occult practice you need to employ to make it work:

echo 'export KEKUS=$(echo "$KEK" | awk -F_ '\''{print $1}'\'')' >> $BASH_ENV

If you know every escape sequence quirk by heart, how cool is that. If you don't, just ask GPT to do it for you or use something else. Since Bash is obsolete by a huge amount, just use Lua or something.
Side note: if you want syntax highlight of YAML inside CircleCI, feel free, unpaid and unrestricted to use this with your YAML outputs, and git diff --color surprisingly works too, otherwise it's a dead horse to ride.

Can't #5: Approval jobs

This section is short but still isn't really documented. While you may be wondering why type has only 1 option, which is approval, be mindful that you cannot have the approval and execute the job too, it will just confirm approval and give you a sweet 404. You are forced to have an actionless job with type: approval and then have an actual job with requires for that approval.

Can't #6: Effectively use conditional steps

This section isn't documented either. Long story short, if you want to use when with steps, anything like $CIRCLE_JOB is considered runtime, and conditions are evaluated at compile time, effectively making you slap those sweet ifs on yet another set of runs you might resort to due to such mayhem.
Side note: when itself, even with convoluted boolean algebra inside it, works as intended. Which, to be honest, is surprising at least.

This should be sufficient to not succumb into insanity while trying to make CircleCI do something more interesting than trivial things. See you next time.

Randych's Abysmal Zealotry