I wanted to know how easy it would be to get Spring-Boot auto-reconfiguration to work in OpenShift. It did not really work for me in Cloud Foundry without some massaging ( auto re-config blog ).

The proper way would be to extend https://github.com/spring-projects/spring-cloud/tree/master/spring-cloud-core for OpenShift Which is probably dead simple, too, but I wanted to do a quick 30 minute hack which become about a 90 minutes dirty hack.

In simple terms, the cartridge always sets -Dspring.profiles.active=cloud (unless this is overwritten by the app developer). And when it finds a ‘OPENSHIFT_MYSQL_DB_URL’ environment variable (which means the user added that service (cartridge) to the application, then the cartridge will add a the MariaDB (MySQL) information to ‘VCAP_SERVICES’ In addition I als set ‘VCAP_APPLICATION’ to some partially correct values, as well as VCAP_APP_HOST and VCAP_APP_PORT.

Then I deployed the jar generated from the unfamous-quotes cloud-foundry-autoreconfig-tests branch and it correctly picked up the MySQL DB and SUCCESS.

It was ridiculously easy to get that working in OpenShift. If I hadn’t put in a typo in one of my first attempts it would have been even easier.

What I did not mentioned in the last blog was that the version of the “unfamous quotes” code only works on Cloud Foundry if there is a data source supplied by Cloud Foundry. Otherwise you get:

org.springframework.cloud.CloudException: No unique service matching interface javax.sql.DataSource found. Expected 1, found 0

This happens due to the this code in DataSourceConfiguration:

return connectionFactory().dataSource();    

With the current version of the spring runner cartridge the same happens: You have to add a MariaDB or it wont work.

Since this was just a quick and dirty test, I did not add any other DB services. But I also would rather check out the spring-core-cloud and create an OpenShift Cloud Connector.

I started with the code from https://github.com/spring-guides/gs-accessing-data-jpa.git (complete). I then added a simple HTML/JS UI on top of the Restful API and added validation.

The way Spring Boot works in this set-up is that it automatically creates an in-memory H2 DB and auto-wires everything together. There is basically no code and it works fine and is simple enough. All in an executable Jar with an embedded Tomcat (~29M).

I uploaded that example to different Cloud PaaS providers as well as private PaaS running in a VM and it
either required no or very little effort to run it (except for one where it failed completely … see previous posts).

What surprised me was that Cloud Foundry does not seem to support auto-reconfig for a jar packaged Spring Boot application. When I added a MySQL service and happily expected it to magically work and instead Cloud Foundry continued to start with the H2 DB (relevant git commit).

I tried different set-ups, but in the end I had to add an additional Spring cloud configuration file:

public class DataSourceConfiguration {
    public static class CloudConfiguration extends AbstractCloudConfig {
        @Bean(destroyMethod = "close")
        public javax.sql.DataSource dataSource() {
            return connectionFactory().dataSource();

And it immediately picked up the MySQL DB Did not create tables, though for that I also had to set spring.jpa.hibernate.ddl-auto=update in the application.property file.

The problem with that approach is that it won’t work locally:

Caused by: org.springframework.cloud.CloudException: No suitable cloud connector found

My thought was that it might just work with using the cloud profile so I changed it

public class DataSourceConfiguration {
    @Profile({ "cloud" })
    public static class CloudConfiguration extends AbstractCloudConfig {
        @Bean(destroyMethod = "close")
        public javax.sql.DataSource dataSource() {
            return connectionFactory().dataSource();

But that profile is not set by Cloud Foundry. One option could be to set the AbstractCloudConfig profile as default and set a different (e.g. local) profile for local development (see DataSourceConfiguration commit)

This works for this project, but might not for others.

Another option could be to set the cloud profile only when deploying to Cloud Foundry via e.g. JAVA_OPTS (see older posts). Either way can be a awkward for anyone not knowing the chosen path.

I prefer the second option because it means that local development works as expected and the same code behaves exactly the same way in Cloud Foundry (ie using H2 instead of MySQL. (see DataSourceConfiguration commit for cloud profile)

And with a specific manifest.yml file this can be overwritten for Cloud Foundry (though I don’t like having this file as part of the code) so that the active Spring Profile is “cloud”. With this set-up the code uses H2 locally and will use the DB service provided by Cloud Foundry – in my case MySQL – when deployed to Cloud Foundry.


Started by making sure that the latest version still works.

I already confirmed with Active State that you cannot upload a JAR directly, but have to unpack it so:

from the working dir (same level as pom.xml)
cd ./unpacked/; unzip ../target/unfamous-quotes-0.0.1-SNAPSHOT.jar; ls -al; cd ..
stackato push –path ./unpacked/

But the application starts up on port 8080 which is not the port stackato wants it to run at and then it is killed and restarted and ..
application.properties still has an entry about the server.port which I commented out and retried. But the server still started on
port 8080. So I made sure that the application.properties file was really updated which it was.

In between tries I always removed application including the route with

stackato delete unfamous-quotes

No luck. nothing helped.
I don’t understand though since I have an example that works and this app worked fine before I did some static file changes.

No matter what I check or do stackato is not able to assign the right port and the server runs on 8080 until it is killed of.
Of course I could set the port like for my simple server test but that defeats the purpose.

Besides, the exact same JAR works on Pivotal’s Cloud Foundation free tier run.pivotal.io with a simple push.

cf push unfamous-quotes -p ./target/unfamous-quotes-0.0.1-SNAPSHOT.jar

Next I tried another trick that I already learned in my experiments. I upload the jar with Eclipse (STS)
Or you can also push with CF, but then I would have to log out of run.pivotal.io – which I am using in parallel –
and into stackato, which also works. But unlike the stackato CLI the cf CLI does not allow multiple targets
and swithcing between them. So Eclispe it is.
The push with Eclipse (or cf) will fail, too, but all it requires is a start of the app and the same jar works in stackato.

Except that this does not resolve anything, the server still tstats on

Setting the JAVA_OPTS environment variable has the same escaping problems as before, too.

At this stage I am ready for one of my once in a month stress-release cigarettes.
The stackato web UI does not do anything to calm me down, either. e.g. the page refresh is annoying.

Even cf set-env did not work I assume there is an incompatibility with the build back

since the logs show this, which neither seems to work nor does it contain any of my environment variables:

stackato[dea_ng.0]: Launching web process: $PWD/.java-buildpack/open_jdk/bin/java -cp $PWD/.:$PWD/.java-buildpack/spring_auto_reconfiguration/spring_auto_reconfiguration-0.8.9.jar -Djava.io.tmpdir=$TMPDIR -XX:MaxPermSize=64M -XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk/bin/killjava.sh -XX:PermSize=64M -Xms382293K -Xmx382293K -Xss995K org.springframework.boot.loader.JarLauncher --server.port=$PORT

And I give up for now.

Initial Set-up

I was able to use an older version of my non-spring test server simple-executable-jar-web-server.

The test is following up on my tests with Cloud Foundry as well as Stackato.

I also created a very simple cartridge (notes) that I was using to understand the way cartridges work. But a
DIY cartridge would have been sufficent for this.

App container cartridge are supposed to come with templates as far as I understand. In this case I packaged the
test server in the template which is also the reason why I wanted it to stay small.

Runtime Environment and Properties


'OPENSHIFT_DEPLOYMENTS_DIR' = '/var/lib/openshift/539/app-deployments/'
'HISTFILE' = '/var/lib/openshift/539/app-root/data/.bash_history'
'OPENSHIFT_SBJR_JAR_DIR' = '/var/lib/openshift/539/app-root/runtime/repo//target/'
'OPENSHIFT_DATA_DIR' = '/var/lib/openshift/539/app-root/data/'
'JAVA_HOME' = '/etc/alternatives/java_sdk_1.6.0'
'PWD' = '/var/lib/openshift/539/spring-boot-jar-runner'
'TMP_DIR' = '/tmp/'
'OPENSHIFT_SECRET_TOKEN' = 'n_3eZmWnvvsJUnY4nYicj-k6vVAG_6nglPrtgkqjyKf0OWH-pGlRbVyCRGRcFAruY4n8BKX7aPKSvZxO-5lQTlWheStOyHkmBcsQ4zwlmJaMjNk9Ii5IkzktC0P9seXS'
'NLSPATH' = '/usr/dt/lib/nls/msg/%L/%N.cat'
'OPENSHIFT_APP_DNS' = 'app1-testapps.openshift.local'
'TMPDIR' = '/tmp/'
'TMP' = '/tmp/'
'OPENSHIFT_SBJR_LOG_DIR' = '/var/lib/openshift/539/spring-boot-jar-runner//logs/'
'PATH' = '/etc/alternatives/java_sdk_1.6.0/bin:/bin:/usr/bin:/usr/sbin'
'OPENSHIFT_SBJR_IDENT' = 'coder_mnm_at:spring-boot-jar-runner:1.0.2:0.0.1'
'OPENSHIFT_CARTRIDGE_SDK_BASH' = '/usr/lib/openshift/cartridge_sdk/bash/sdk'
'OPENSHIFT_BUILD_DEPENDENCIES_DIR' = '/var/lib/openshift/539/app-root/runtime/build-dependencies/'
'OPENSHIFT_SBJR_DIR' = '/var/lib/openshift/539/spring-boot-jar-runner/'
'OPENSHIFT_GEAR_DNS' = 'app1-testapps.openshift.local'
'OPENSHIFT_REPO_DIR' = '/var/lib/openshift/539/app-root/runtime/repo/'
'SHLVL' = '3'
'XFILESEARCHPATH' = '/usr/dt/app-defaults/%L/Dt'
'OPENSHIFT_APP_SSH_PUBLIC_KEY' = '/var/lib/openshift/539/.openshift_ssh/id_rsa.pub'
'LOGNAME' = '539'
'OPENSHIFT_SBJR_JAR_BASE_DIR' = '/var/lib/openshift/539/app-root/runtime/repo/'
'OPENSHIFT_PRIMARY_CARTRIDGE_DIR' = '/var/lib/openshift/539/spring-boot-jar-runner/'
'_' = '/bin/java'
'OPENSHIFT_CLOUD_DOMAIN' = 'openshift.local'
'OPENSHIFT_DEPENDENCIES_DIR' = '/var/lib/openshift/539/app-root/runtime/dependencies/'
'SHELL' = '/bin/sh'
'OPENSHIFT_SBJR_LOG' = '/var/lib/openshift/539/spring-boot-jar-runner//logs/main-SBJR-log'
'OPENSHIFT_HOMEDIR' = '/var/lib/openshift/539/'
'GEM_HOME' = '/var/lib/openshift/539/.gem'
'OPENSHIFT_APP_SSH_KEY' = '/var/lib/openshift/539/.openshift_ssh/id_rsa'
'OPENSHIFT_CARTRIDGE_SDK_RUBY' = '/usr/lib/openshift/cartridge_sdk/ruby/sdk.rb'
'USER' = '539'
'HOME' = '/var/lib/openshift/539'
'OPENSHIFT_SBJR_JDK6' = '/etc/alternatives/java_sdk_1.6.0'
'OPENSHIFT_SBJR_JDK7' = '/etc/alternatives/java_sdk_1.7.0'
'OPENSHIFT_SBJR_PATH_ELEMENT' = '/etc/alternatives/java_sdk_1.6.0/bin'

System Properties

'java.runtime.name' = 'OpenJDK Runtime Environment'
'sun.boot.library.path' = '/usr/lib/jvm/java-1.7.0-openjdk-'
'java.vm.version' = '24.45-b08'
'java.vm.vendor' = 'Oracle Corporation'
'java.vendor.url' = 'http://java.oracle.com/'
'path.separator' = ':'
'java.vm.name' = 'OpenJDK 64-Bit Server VM'
'file.encoding.pkg' = 'sun.io'
'user.country' = 'US'
'sun.java.launcher' = 'SUN_STANDARD'
'sun.os.patch.level' = 'unknown'
'server.port' = ''
'java.vm.specification.name' = 'Java Virtual Machine Specification'
'user.dir' = '/var/lib/openshift/539/spring-boot-jar-runner'
'java.runtime.version' = '1.7.0_45-mockbuild_2013_10_16_17_47-b00'
'java.awt.graphicsenv' = 'sun.awt.X11GraphicsEnvironment'
'java.endorsed.dirs' = '/usr/lib/jvm/java-1.7.0-openjdk-'
'os.arch' = 'amd64'
'java.io.tmpdir' = '/tmp'
'line.separator' = '
'java.vm.specification.vendor' = 'Oracle Corporation'
'os.name' = 'Linux'
'sun.jnu.encoding' = 'ANSI_X3.4-1968'
'java.library.path' = '/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib'
'java.specification.name' = 'Java Platform API Specification'
'java.class.version' = '51.0'
'sun.management.compiler' = 'HotSpot 64-Bit Tiered Compilers'
'os.version' = '3.11.10-200.fc19.x86_64'
'user.home' = '/var/lib/openshift/539'
'user.timezone' = 'US/Pacific'
'java.awt.printerjob' = 'sun.print.PSPrinterJob'
'file.encoding' = 'ANSI_X3.4-1968'
'java.specification.version' = '1.7'
'server.address' = ''
'java.class.path' = '/var/lib/openshift/539/spring-boot-jar-runner/web-app-jar/simple-executable-jar-web-server-0.0.1.jar'
'user.name' = '539'
'java.vm.specification.version' = '1.7'
'sun.java.command' = '/var/lib/openshift/539/spring-boot-jar-runner/web-app-jar/simple-executable-jar-web-server-0.0.1.jar'
'java.home' = '/usr/lib/jvm/java-1.7.0-openjdk-'
'sun.arch.data.model' = '64'
'user.language' = 'en'
'java.specification.vendor' = 'Oracle Corporation'
'awt.toolkit' = 'sun.awt.X11.XToolkit'
'java.vm.info' = 'mixed mode'
'logging.path' = '/var/lib/openshift/539/spring-boot-jar-runner//logs/'
'java.version' = '1.7.0_45'
'java.ext.dirs' = '/usr/lib/jvm/java-1.7.0-openjdk-'
'sun.boot.class.path' = '/usr/lib/jvm/java-1.7.0-openjdk-'
'java.vendor' = 'Oracle Corporation'
'file.separator' = '/'
'java.vendor.url.bug' = 'http://bugreport.sun.com/bugreport/'
'sun.io.unicode.encoding' = 'UnicodeLittle'
'sun.cpu.endian' = 'little'
'sun.cpu.isalist' = ''
Here is what I know about you :
Your adress:
Your request headers:
'X-forwarded-proto' = '[http]'
'X-request-start' = '[t=1402811035856892]'
'Host' = '[app1-testapps.openshift.local]'
'X-forwarded-server' = '[broker-66587a.openshift.local]'
'Accept-encoding' = '[gzip,deflate,sdch]'
'X-forwarded-port' = '[80]'
'Connection' = '[close]'
'X-forwarded-for' = '[]'
'X-client-ip' = '[]'
'Accept-language' = '[en-GB,en-US;q=0.8,en;q=0.6,de;q=0.4]'
'X-forwarded-host' = '[app1-testapps.openshift.local]'
'User-agent' = '[Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36]'
'Accept' = '[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8]'

Adding MariaDB

From the three experiences I found the OpenShift one the friendliest in terms of adding a new service as well as the information that I got out of it.
It also immediately showed the connection details to the DB.
Only a restart is required, not a full push.

MySQL 5.1 database added.  Please make note of these credentials:

    Root User: admin123
Root Password: passwd123
Database Name: app1


You can manage your new MySQL database by also embedding phpmyadmin.
The phpmyadmin username and password will be the same as the MySQL credentials above.

Yep, it is MySQL.

A lot more variables changed in comparison to the other two and no system properties:

> 'OPENSHIFT_MARIADB_DIR' = '/var/lib/openshift/539/mariadb/'
> 'OPENSHIFT_MARIADB_DB_SOCKET' = '/var/lib/openshift/539/mariadb//socket/mysql.sock'
> 'OPENSHIFT_MYSQL_DB_URL' = 'mysql://admin123:passwd123@'
> 'OPENSHIFT_MYSQL_DB_LOG_DIR' = '/var/lib/openshift/539/mariadb//log/'
> 'OPENSHIFT_MARIADB_IDENT' = 'redhat:mariadb:5.5:0.2.4'
> 'OPENSHIFT_MARIADB_DB_LOG_DIR' = '/var/lib/openshift/539/mariadb//log/'
> 'OPENSHIFT_MARIADB_DB_URL' = 'mysql://admin123:passwd123@'
> 'OPENSHIFT_MYSQL_DB_SOCKET' = '/var/lib/openshift/539/mariadb//socket/mysql.sock'

Similar to Stackato there is a lot of duplication, in both cases I assume the reason is backward compatibility.

Adding MongoDB and PostgreSQL as well

MongoDB 2.2 database added.  Please make note of these credentials:

Root User:     adminuser
Root Password: passwd123
Database Name: app1


PostgreSQL 9.2 database added.  Please make note of these credentials:

    Root User: adminuser
    Root Password: passwd123
    Database Name: app1


changes are :

> 'MANPATH' = ''
> 'OPENSHIFT_MONGODB_DB_LOG_DIR' = '/var/lib/openshift/539/mongodb//log/'
> 'OPENSHIFT_MONGODB_DB_URL' = 'mongodb://adminuser:passwd123@'
> 'OPENSHIFT_MONGODB_DIR' = '/var/lib/openshift/539/mongodb/'
> 'OPENSHIFT_MONGODB_IDENT' = 'redhat:mongodb:2.2:0.2.5'
> 'OPENSHIFT_POSTGRESQL_DB_LOG_DIR' = '/var/lib/openshift/539/postgresql//log/'
> 'OPENSHIFT_POSTGRESQL_DB_PID' = '/var/lib/openshift/539/postgresql//pid/postgres.pid'
> 'OPENSHIFT_POSTGRESQL_DB_SOCKET' = '/var/lib/openshift/539/postgresql//socket/'
> 'OPENSHIFT_POSTGRESQL_DB_URL' = 'postgresql://adminuser:passwd123@'
> 'OPENSHIFT_POSTGRESQL_DIR' = '/var/lib/openshift/539/postgresql/'
> 'OPENSHIFT_POSTGRESQL_IDENT' = 'redhat:postgresql:9.2:0.3.5'
> 'PGDATABASE' = 'app1'
> 'PGDATA' = '/var/lib/openshift/539/postgresql//data'
> 'PGHOST' = '/var/lib/openshift/539/postgresql//socket/'
> 'PGUSER' = 'adminuser'

The Basics

I used the DIY cartridge before and played with it a bit. Next I wanted to create a custom cartridge. As usual for me the first attempt was more about learning about the system than about getting things to work right away. And I must say it was a bit annoying when you start with zero or little knowledge.

A working version is here : https://bitbucket.org/markus_ebenhoeh/openshift-cartridge-spring-boot-jar-runner

Learn from my mistakes:

  • make sure that you use the right cartridge version. I followed code that was newer than the version I was running, but even then I mixed old with new conventions. E.g. I used OPENSHIFT_LOG_DIR but that does not exist yet.
  • don’t git push your code, then find the manifest URL and then try to test it. Either use the card reflector (usage) or if you already have an http server running, just serve your code directly I changed the source to a zip on my http server and ran this just before I retried with a new application rm -f cartridge-name.zip; zip -r cartridge-name.zip *; chmod a+rX -R $PWD In the end, I just mounted a shared folder and referred the source URL to that with file:///mnt/public-shared/openshift-cartridge-spring-boot-jar-runner while still hosting the same folder via http for the manifest access

  • to test my cartridge I added a spring-boot application jar with an embedded tomcat. Instead of taking the smallest web-app jar that I could find I took a 29MB jar and added it into the template. This is stupid on many levels, but it also meant that it took long to do the git pull and the start and the transfer.etc having a test app in the template is important for the testing of the cartridge and afterwards for the use of it, but a smaller application is what I am going for next.

Debugging a Failing Cartridge

When my cartridge completely failed I had no output and nothing to go on. I was logged on to my test origin VM but the gear/application folder would simple be deleted at the end. So all my test log output that was supposed to show me were it failed and what the environment looked like would vanish

So while being logged on to the test VM with root I did this dodgy thing:

cd /var/lib/openshift/
# my user/gear/app folder  always starts with 5 and I had no other gears set-up
unalias cp
while true; do cp -a 5* tmp/ ;  usleep 100000; echo next  ;done

once it failed I had more information to go on with. especially the nev folders are useful

Other problems:

  • I would like to see a cartridge life-cycle flow. e.g. copy file, call install, call setup etc.
  • when are ENV variables available? Can I refer to my own env_xyz.erb variables when defining new ones?
  • I still have no idea how the repo
  • there has to be a better way to test this then to do an add-app with the updated cartridge.
  • I could not find any debug output when my cartridge completely failed
  • CDK is not working on Origin bug

OpenShift Cartridge Development Kit

ran this against OpenShift Online since CDK is not working on Origin bug. not sure what the point is, though. I think it is about being able to have a simple way of hosting the cartridge for a certain version

rhc create-app mycart http://cdk-claytondev.rhcloud.com
The cartridge 'http://cdk-claytondev.rhcloud.com' will be downloaded and installed

Application Options
Domain:     testcdk
Cartridges: http://cdk-claytondev.rhcloud.com
Gear Size:  default
Scaling:    no

Creating application 'mycart' ... done

The CDK is configured with password XXXXXXXXXXXXXXX for builds (use 'admin' as the user)

Waiting for your DNS name to be available ... done

Cloning into 'mycart'...
The authenticity of host 'mycart-testcdk.rhcloud.com (' can't be established.
RSA key fingerprint is cf:ee:77:cb:0e:fc:02:d7:72:7e:ae:80:c0:90:88:a7.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mycart-testcdk.rhcloud.com,' (RSA) to the list of known hosts.
X11 forwarding request failed on channel 0

Your application 'mycart' is now available.

URL:        http://mycart-testcdk.rhcloud.com/
SSH to:     539@mycart-testcdk.rhcloud.com
Git remote: ssh://539@mycart-testcdk.rhcloud.com/~/git/mycart.git/
Cloned to:  /xdata/local/dev/markus/git/cdk/mycart

Run 'rhc show-app mycart' for more details about your app.

Lifeccyle Notes

first installation

20140609_034324 setup called with --version 1.0.2 
20140609_034324 install called with --version 1.0.2 
20140609_034324 control called with start 
20140609_034324 start_app called 
20140609_034324 Starting spring-boot-bin-jar-runner cartridge 
20140609_034324 start_app: Using jarfile=/var/lib/openshift/539/spring-boot-jar-runner/web-app-jar/simple-executable-jar-web-server-0.0.1.jar 
20140609_034324 control : checkSpringBootAppIsRunning still checking 
20140609_034325 control start successfully started the application 

git push new app code

20140609_034411 control called with stop 
20140609_034411 stop_app called 
20140609_034411 Stopping spring-boot-bin-jar-runner cartridge 
20140609_034411 Sending SIGTERM to java:4556 ... 
20140609_034411 control called with pre-repo-archive 
20140609_034412 control called with update-configuration 
20140609_034412 control called with pre-build 
20140609_034412 control called with build 
20140609_034412 build called with 
20140609_034412 control called with update-configuration 
20140609_034412 control called with deploy 
20140609_034412 deploy called with 
20140609_034412 control called with start 
20140609_034412 start_app called 
20140609_034412 Starting spring-boot-bin-jar-runner cartridge 
20140609_034412 start_app: Using jarfile=/var/lib/openshift/539/spring-boot-jar-runner/web-app-jar/simple-executable-jar-web-server-0.0.1.jar 
20140609_034412 control : checkSpringBootAppIsRunning still checking 
20140609_034412 control start successfully started the application 
20140609_034432 control called with post-deploy 

There is considerable time spent between “start” and “post-deploy” not sure what’s happening there.