How I Learned to Love the Lack of Environment Variables

Previously I posted a preview of my favorite new feature of OpenStackClient, support for cloud configuration files and the new --os-cloud global option. Who knew it would take six months to get it done and released? There is some merely adequate documentation of the Client Cloud Configuration (or CCC) operation in the OSC Configuration docs, but this is too good to leave to official docs alone.

CCC was actually introduced in OSC release 1.1.0, which is the basis for compatibility with Liberty dependency requirements. But like the rest of OSC, it works just fine with all of the supported OpenStack releases.

* the current directory
* ``~/.config/openstack``, aka ``$XDG_CONFIG_HOME/openstack``
* ``/etc/openstack``

clouds:
  devstack:
    auth:
      auth_url: http://127.0.0.1:35357/
      project_name: demo
      username: demo
      password: 0penstack
    region_name: RegionOne

openstack --os-cloud devstack token issue

public-clouds:
    hp:
        auth:
            auth_url: https://region-b.geo-1.identity.hpcloudsvc.com:35357/v2.0
        region_name: region-b.geo-1
        dns_service_type: 'hpext:dns'
        image_api_version: 1
        image_format: qcow2
    rackspace:
        auth:
            auth_url: https://identity.api.rackspacecloud.com/v2.0/
        database_service_type: 'rax:database'
        compute_service_name: cloudServersOpenStack
        image_api_version: 2
        image_api_use_tasks: True
        image_format: vhd
    dreamhost:
        auth:
            auth_url: https://keystone.dream.io/v2.0
        region_name: RegionOne
        image_api_version: 2
        image_format: raw

clouds:
    rax:
        cloud: rackspace
        auth:
            project_id: 123456
            username: joe-stacker
            password: 0penstack
        region_name: DFW

openstack --os-cloud rax server list

OS_CLOUD=rax openstack server list

Followup

Oh, just one more thing... I promised a followup to this post once things settled down. I am settling into my new role with Intel doing pretty much the same set of upstream things I was doing before. As hoped, OpenStackClient is a primary focus, along with some of the other client-side and app developer things I have been peripherally involved in lately like the Python SDK and the API Working Group. And DevStack. Always DevStack.

Actually, two more things... I was also elected to the OpenStack Technical Committee in the recent election.

How about one more one more thing? I shamelessly grabbed control of this dormant repo and intend to keep slowly plugging away at recreating a Keystone Session-style client in my favorite strongly-typed language. Come join!

Looking Backward

I don't want to spend too much energy looking at where we have been, except to note that a lot has happened since I shared a cab to SFO with Dolph Matthews and I floated the idea of scrapping all of the OpenStack project CLIs and starting over. He was just receptive enough that I spent most of the flight home teasing out the basic set of command actions and objects. And he backed that up by later choosing to not implement a CLI for the Identity v3 API but instead use OSC.

The rigorous command structure is what I consider the big win for OSC, having a small set of common actions and a set of known objects that are manipulated with those actions. Define a common operation and then do that operation on EVERY object the same way, even if that means a bunch of work under the hood because the REST API doesn't work that way.

One now-forgotten tidbit here is that I designed the commands to look similar to VMS's DCL command line set, in that the action preceeds the object. One complaint was that the objects needed to be pluralized on some actios, like list, because that's the proper English thing to do. I ignored plurals completely, although it does seem like we could have done it only with the list command.

It was at the Portland summit that I was persuaded to make the change when someone who understands bash command completion better than I do (which is to say 'I never looked at it') mentioned that it would be much simpler if the object came first. OK, finally, a good technical reason to change so we did. I think dhellmann said something like "I expected that to be much harder" afterward. I do change my mind, sometimes.

Looking Around Today

In the three years we have had just over 1000 commits from around 70 contributors. I want to grow the team; we have a small band of regulars now with a core review team of 4, having recently added Terry Howe, who has also spent a significant amount of time working on the Python SDK.

We have a couple of interesting features brewing for the next significant release, support for the os-client-config cloud configuration files and a client-side caching that will significantly improve responsiveness in many instances. [2]

Another historical bit is that for most of its life, OSC was mostly a side project for me. I was able to eventually give it some real time along side DevStack and the other things but it seems to be the kind of project that doesn't seem so important until you really really need it now.

As I was talking to a lot of people over the last couple of weeks trying to find my next professional home, it became clear to me how some people and organizations feel about projects like OSC. They're not sexy, not the sort of thing that 'sells' customers on your cloud or your service or whatever. There were a couple of companies that totally did not view this as worth spending time on. Fortunately for us all, there are more companies that do think it is worth the time and I expect to be able to give OSC and other client/app developer projects a significant part of my time going forward.

When OSC was added as an official project, we also brought the os-client-config (from Stackforge) and cliff (from Oslo) repos with it. Cliff was written by Doug Hellmann specifically for OSC to manage the numerous command implementations and give a solid basis for each of the actions. I think it has worked out great and been a great help in maintaining the consistency in OSC so far. It has also been adopted by other projects, some outside of the OpenStack ecosystem.

os-client-config (aka o-c-c) was originated by Monty Taylor to add client-side cloud configuration along with his shade library. BTW, if you ever want to see what the challenges of using multiple clouds in the same project, have a look at shade. In an ideal world it should not need to exist, especially since it only talks to OpenStack clouds.

OSC will be using o-c-c to implement the same cloud configuration that will make it simple to switch between cloud authentication configurations with a single command line option. And also to share public cloud configuration templates without sharing authentication details for them. I plan to follow up with details on how to use this soon, watch this space.

Looking Ahead

Technically we still have a lot of work to do. There are significant APIs that are still not implemented (Network, Volume v2) and incomplete (Image v1 and v2). We need to speed up the load time, eliminate unnecessary dependencies, and fix bugs. Always fix bugs.

On a slightly longer scope, we will switch to use the OpenStack SDK when it is ready, continue to better enable plugins to support upper layer services, and most importantly, we MUST drive down the number of dependencies required to install OSC and make it easy to use for non-Python developers. There needs to be a single file install that 'just works'.

I do still want to duplicate it all in Go. My Go prototype is older than OSC, actually...and is a single file install...

DevStack

There are two major priorities for DevStack: run services in virtual environments and use Neutron as the default network stack.

Grenade

The Grenade plugin story has not been completed, it is in the first set of draft rewrites to remove the zombies and vampires and make it just a 'plugin exiled to project repo' kind of story.

There were a pair of minor reviews landed to support the DevStack/Neutron rework but they demanded more lines and were relegated to the 'C' story that parallels Rosencrantz and Guildenstern, who of course are now dead.

osc-debug

I've spent a lot of time debugging OSC lately in preparation for my favorite new feature yet-to-come, support for the new --os-cloud option that takes advantage of os-client-config's cloud configuration file abilities. Much of this debugging is necessarily around authentication and it became obvious to me that a built-in way to see what OSC was using for authentication might be handy and osc-debug was born.

As far as plugins go, osc-debug is pretty ordinary. It just adds three new commands (so far) that dump internal OSC state information.

osc api list
    [--type <service-type>]
    [--name <service-name>]
    [--long]

osc auth show

osc auth type list

osc auth type show
    <auth-type>

osc-quintette

osc-quintette is where the experimentation really gets going. This oddly-named [1] plugin has a couple of things, the most interesting one is how the server create command has been extended to accept --ram, --disk and --vcpu arguments to auto-select a matching flavor for the new server.

While the functionality is nice, the interesting bit is how the stock commands are extended.

osc flavor find
    [--ram <size-mb>]
    [--disk <size-gb>]
    [--vcpus <vcpus>]

osc server create
    [--ram <size-mb>]
    [--disk <size-gb>]
    [--vcpus <vcpus>]
    ...
    <server-name>

What Else?

Imagination is all that is required to go from here. Go forth and plugin.

local.conf

This is DevStack's configuration file. It will never be modified by DevStack.

LOGDIR=$DEST/logs
LOGFILE=$LOGDIR/stack.sh.log  # why didn't bare name work???

FIXED_RANGE=10.254.1.0/24

enable_service dstat
disable_service h-eng h-api h-api-cfn h-api-cw
disable_service horizon
#enable_service s-proxy s-object s-container s-account

# Nova Net
enable_service n-net
disable_Service q-svc q-agt q-dhcp q-l3 q-meta

# Neutron
# NETWORK_GATEWAY must be in the FIXED_RANGE network
#NETWORK_GATEWAY=10.254.1.1
#disable_service n-net
#enable_service q-svc q-agt q-dhcp q-l3 q-meta

disable_service n-obj n-crt

[[post-config|$NOVA_CONF]]
[DEFAULT]
api_rate_limit = False

But First, Preview Time

There is one additional yet-to-come feature that I can't resist mentioning now that it has been proposed for review. It leverages mordred's os-client-config module to read configuration information from a file by name. In plain language, rather than set up a handful of environment variables or command-line options, all of the authentication and other configuration for OSC can be stashed in a YAML file and called by name:

openstack --os-cloud devstack-1 image list --long

This is also step one in simplifying dealing with multiple clouds:

for cloud in devstack-1 hpcloud-az2 rax-ord; do
    openstack --os-cloud image list --long

It is a small thing, but small things often make us happy. It figures in to the following authentication discussion that I don't want to update again in a month. So until os-cloud support merges, ignore references to YAML, CloudConfig, etc. below.

Sources of Truth

OpenStackClient has three sources of configuration information (in decreasing priority order):

• command line options
• environment
• CloudConfig (~/.config/openstack/clouds.yaml file)

Once all of the sources have been processed and a single configuration object assembled, the fun can begin. If an authentication type is not provided, the authentication options are examined to determine if one of the default types can be used. If no match is found an error is reported and a period of stillness is declared. Rather, the program exits.

Note that the authentication call to the Identity service has not yet occurred. It is deferred until the last possible moment in order to reduce the number of unnecessary queries to the server, such as when further processing detects an invalid command.

Keystone Authentication Plugins

The Keystone client library implements the base set of plugins. Additional plugins may be available from the Keystone project or other sources. See the Keystone client documentation for more information.

There are at least three authentication types that are always available:

• Password: A username and password, plus optional project and/or domain, are used to identify the user. This is the most common type and the default any time a username is supplied. An authentication URL for the Identity service is also required. [Required: --os-auth-url, --os-project-name, --os-username; Optional: --os-password]

• Token: This is slightly different from the usual token authentication (described below as token/endpoint) in that a token and an authentication URL are supplied and the plugin retrieves a new (scoped?) token. [Required: --os-auth-url, --os-token]

• Token/Endpoint: This is the original token authentication (known as 'token flow' in the early CLI documentation in the OpenStack wiki). It requires a token and a direct endpoint that is used in the API call. The difference from the new Token type is this token is used as-is, no call is made to the Identity service from the client. This type is most often used to bootstrap a Keystone server where the token is the admin_token configured in keystone.conf. It will also work with other services and a regular scoped token such as one obtained from a token issue command. [Required: --os-url, --os-token]

  [Note that the Token/Endpoint plugin is currently supplied by OSC itself and is not available for other clients using the Keystone client lib. It shall move to its Proper Home in Good Time.]

• Others: There are SAML and other (Kerberos?) plugins under development that are also supported. To use them they must be selected by supplying the --os-auth-type options.

How It's Made

[Who doesn't love that show? ]

The authentication process flows from OSC's OpenStackShell to the New-and-Improved ClientManager.

• But first, on import api.auth:
  • obtains the list of installed Keystone authentication plugins from the keystoneclient.auth.plugin entry point.
  • builds a list of authentication options from the plugins.
• OpenStackShell parses the command line:
  • If --os-cloud is present read the named configuration from ~/.config/openstack/clouds.yaml and create a CloudConfig object
    • CloudConfig also handles picking up the matching environment variables for the options
  • The remaining global command line options are merged into the new CloudConfig
• A new ClientManager is created and provided with the CloudConfig:
  • If --os-auth-type is provided and is a valid and available plugin it is used.
  • If --os-auth-type is not provided select an authentication plugin based on the existing options. This is a short-circuit evaluation, first match wins.
    • If --os-endpoint and --os-token are both present token_endpoint is selected
    • If --os-username is present password is selected
    • If --os-token is present token is selected
    • If no selection has been made by now exit with error
  • Load the selected plugin class.
• ClientManager waits until an operation that requires authentication is attempted to make the initial request to the Identity service.
  • if --os-auth-url is not present for any of the types except Token/Endpoint, exit with an error.

Destinations of Consequences

The changes that began with utilizing Keystone client's Session are nearly complete and have added the openstackclient.api.auth module and drastically restructured the openstackclient.shell and openstackclient.clientmanger modules. One result is that the ClientManager is now nearly self-contained with regard to its usability apart from the OSC shell. At this time I can neither confirm nor deny that ClientManager could be used as a single-point client API. While it works (one example) it is not yet a stable API because it only unifies the session and auth components, passing the real work down to either the project libraries or OSC's internal API objects. So don't go and do that. Yet...

Layer 1

We're still the one...

We open with the addition of Neutron as a viable alternative to Nova Network, and the likelihood of it becoming the default configuration in DevStack early in the Juno cycle.

• Identity (Keystone)
• Image (Glance)
• Network (Neutron)
• Compute (Nova)

What really stands out to me now is the realization that all of these were originally part of Nova itself (plus the cinder volume service, more on that later). They were broken apart or re-implemented to scale the development as Nova kept growing. In fact, there is talk again of need to break out more simply because Nova continues to expand.

This is the smallest working set for a compute-enabled cloud. Realistic useful clouds of course offer more than this, so we have...

Layer 2

Layer 2 services are optional in a useful compute cloud but some are also useful in their own right as non-compute cloud services.

So the current Layer 2 still contains:

• Volume (Cinder)
• Object (Swift)
• Bare-metal (Ironic)

These all build on the Layer 1 nucleus and get us a practical useful cloud. They also all have the characteristic of having dependency arrows pointing out of Layer 1 when used with a compute cloud, such as Glance using Swift as its backend store. This is a defining characteristic that brings a project in to Layer 2.

Even though Cinder was literally carved out of the Nova code base it stays in Layer 2 because it is an optional service to a Layer 1 cloud. Manila will also fit here for the same reasons.

I neglected to mention last time the ability of Swift to stand alone as a useful cloud service as it has maintained its own authentication capability. However, using it with any other OpenStack services requires Swift to use Keystone.

I also think it is worth considering the direction of the trademark usage constraints the board is refining with the DefCore work. The current DefCore capability proposal is satisfied using only Layer 1 and 2 projects. Also, the stand-alone services currently would not be able to qualify for trademark usage when deployed alone.

Layer 3

Do something useful. Host services for paying customers. Provide Lego blocks for them to build awesome cloud apps. Warn about runaway while true; done loops. Count cycles burned and bits sent so paying customers know what to pay. Communicate with your useful cloud.

The rest of the OpenStack-affiliated projects (for some value of affiliated) go in Layer 3 to populate the big tent. If we've done our job right the majority of everything else should be able to be accomplished without special consideration from Layers 1 and 2. Broad categories of Layer 3 projects include:

• User Interfaces - You need one but a feature of well documented REST APIs is allowing the client side to be easily replaceable.
  • Orchestration (Heat) (it is basically a smart automated cloud client, no?)
  • Web UI (Horizon)
  • <insert-one-of-the-other-CLI-or-web-clients-here>
• Something-as-a-Service - These are all services a deployer may choose to offer.
  • Database (Trove)
  • Message Passing (Zaqar)
• Tracking Snooping and Counting - Keeping an eye on the useful cloud
  • Telemetry (Ceilometer)

Why is Heat in Layer 3??? Heat is essentially a smart automated cloud client and should be treated as one. It needs to meet the same requirements for API compatibility to be useful over time.

Layer 4

Layer 4 is everything else that is not OpenStack-affiliated but might be a part of an especially useful OpenStack cloud. Things like Ceph or Apache jclouds are useful with and as part of OpenStack clouds, but they also have a life of their own and we should respect that and not call late at night.

What About Layer 0?

Ah, right, where the Libraries live. The last year has seen significant changes to how OpenStack-related libraries are integrated with a number of Oslo libraries being released stand-alone. In most cases these can and should be thought of as dependencies just as any non-OpenStack project dependency (like SQLAlchemy or requests) that happen to live in either the openstack or stackforge namespaces in our Git repositories.

It also seems appropriate to add the client API libraries and SDKs to Layer 0 as the dependency model and release schedule is very similar to the other libraries. I am specifically not including command-line interfaces here as I think those belong in Layer 3 but the project client libraries have an embedded CLI so the'll straddle the boundaries no matter what.

So How Do We Govern and Test All This?

OK, I lied. I said I would skip this, but anyone still reading must think this is has a thread of merit, right? I choose to make that assumption going forward, and those of you still reading for a laugh, here is your cue.

I'll lay out an overview of a developers perspective because I am primarily an OpenStack developer and I need the world to know what I think. However, I am also an application developer and cloud end-user so those perspectives are not lost. I have not managed to add cloud deployer to my CV, yet.

Horizontal Projects

While it feels like I'm saving the best for last, in reality much of the above has to have some structure to know the scope that Infrastructure, Docs and QA need to be able to support. Focusing these on Layers 1 and 2 provides a clear limit to the scope required. This is not to say that other projects are not going to be accommodated, particularly those already in the current release, but it does say that it is not assured.

Now What Smart Guy?

With my thoughts on Layers updated to include the governance and testing considerations it is time to match up other perspectives, flesh out the above with the new information and catch up on the plethora of other posts on this topic.

Film at eleven...

api.BaseAPI

We start with a BaseAPI object that contains the common operations. It is pretty obvious there are only a couple of ways to get a list of resources from OpenStack APIs so the bulk of that and similar actions are here.

It is also very convenient to carry around a couple of other objects so they do not have to be passed in every call. BaseAPI contains a session, service type and endpoint for each instance. The session is a requests.session.Session-compatible object. In this implementation we are using the keystoneclient.session.Session which is close enough. We use the ksc Session to take advantage of keystoneclient's authentication plugins.

The service type and endpoint attributes are specific to each API. service type is as it is used in the Service Catalog, i.e. Compute, Identity, etc. endpoint is the base URL extracted from the service catalog and is prepended to the passed URL strings in the API method calls.

Most of the methods in BaseAPI also are meant as foundational building blocks for the service APIs. As such they have a pretty flexible list of arguments, many of them accepting a session to override the base session. This layer is also where the JSON decoding takes place, these all return a Python list or dict.

The derived classes from BaseAPI will contain all of the methods used to access their respective REST API. Some of these will grow quite large...

api.object_store.APIv1

While this is a port of the existing code from OpenStackClient, object_store.APIv1 is still essentially a greenfield implementation of the Object-Store API. All of the path manipulation, save for prepending the base URL, is done at this layer.

api.compute.APIv2

This is one of the big ones. At this point, only flavor_list(), flavor_show() and key_list() have been implemented in compute.APIv2.

Unlike the object-store API, the rest of the OpenStack services return resources wrapped up in a top-level dict keyed with the base name of the resource. This layer shall remove that wrapper so the returned values are all directly lists or dicts. This removed the variations in server implementations where some wrap the list object individually and some wrap the entire list once. Also, Keystone's tendency to insert an additional values key into the return.

api.identity_vX.APIvX

The naming of identity_v2.APIv2 and identity_v3.APIv3 is a bit repetitive but putting the version into the module name lets us break down the already-long files.

At this point, only project_list() is implemented in an effort to work out the mechanics of supporting multiple API versions. In OSC, this is already handled in the ClientManager and individual client classes so there is not much to see here. It may be different otherwise.

OSC Usage

To demonstrate how this API is used, I've added an BaseAPI instance to the existing client objects that get stored in the ClientManager. For example, the addition for compute.client is one object instantiation and an import. Now in OSC, clientmanager.compute.api has all of the (implemented) Compute API methods.

Using it in the flavor commands is a simple change to call compute.api methods rather than the compute.flavor.XXX methods.

Setting up for multiple API versions took a bit more work, as shown in identity.client. A parallel construction to the client class lookup is required, and would totally replace the existing version lookup once the old client is no longer required.

Fluff

One other cool feature is utilizing requests_mock for testing from the start. It works great and has not the problems that rode along with httpretty.

Now What?

Many object models could be built on top of this API design. The API object hierarchy harkens back to the original client lib Manager classes, except that they encompass an entire REST API and not one for each resource type.

But You Said 'Sanity' Earlier!

Sanity in terms of coalescing the distinct APIs into something a bit more common? Yes. However, this isn't going to fix everything, just some of the little things that application developers really shouldn't have to worry about. I want the project REST API docs to be usable, with maybe a couple of notes for the differences.

For example, OSC and this implementation both use the word project in place of tenant. Everywhere. Even where the underlying API uses tenant. This is an easy change for a developer to remember. I think.

Also, smoothing out the returned data structures to not include the resource wrappers is an easy one.

Duplicating Work?

"Doesn't this duplicate what is already being done in the OpenStack Python SDK?"

Really, no. This is meant to be the low-level SDK API that the Resource model can utilize to provide the back-end to its object model. Honestly, most applications are going to want to use the Resource model, or an even higher API that makes easy things really easy, and hard things not-so-hard, as long as you buy in to the assumptions baked in to the implementation.

Sort of like OS/X or iOS. Simple to use, as long as you don't want to anything different. Maybe we should call that top-most API iOSAPI?