My 2 days Journey on moving services to a swarm

<p class="wp-block-paragraph">I work for starsellersworld.com, I wanted my company start to use tool for tracking project development in a centralized way, Jira was expensive as resource, and I can not ask such a step, so I asked a machine were to run <strong>Redmine</strong>. Then it cames the <strong>gogs</strong>, and, after little time, <strong>Jenkins</strong>.</p> <p class="wp-block-paragraph">A number of tool I never used before, I am a developer, not a DevOps, I install those as docker containers. Containers communicate each other over internal network, but there was 3 different docker network, so I added the network to each container and specify its address by IP. That strange configuration worked for more than a year. Then it was becoming frustating, I decided to spare time to switch to swarm mode.</p> <figure class="wp-block-image"><img src="https://smartango.com/wp-content/uploads/2019/06/NetworkTopology-FullyConnected.png" alt="" class="wp-image-339"/></figure> <h3 class="wp-block-heading">Porting data</h3> <p class="wp-block-paragraph">Problem of porting data to containers managed by the docker swarm are related mostly to data about references between them: the IP must be replaced by the service’s name.</p> <p class="wp-block-paragraph"><span style="text-decoration: underline;">Redmine</span>. Using <span style="text-decoration: underline;">gogs</span> repository, redmine refer to its via IP, there is no way to edit the repository configuration into redmine, the only option, via webgui, is to remove and recreate that link. The solution was to enter into the postgresql db and update the repository address with a query directly:</p> <p class="wp-block-paragraph">$ docker exec -it redmine redmine.1.[hashcodehere] bash<br> psql -U redmine -d redmine_production</p> <pre class="wp-block-preformatted">UPDATE repositories SET extra_info = regexp_replace(extra_info, '^(.*)extra_clone_url: ssh:\/\/git@(172.17.0.2:22)(.*)$', '\1extra_clone_url: ssh://git@gogs:22\3', 'g')</pre> <p class="wp-block-paragraph"><span style="text-decoration: underline;">Jenkins</span>. This also refers to gogs for the repo retrievement by IP, and must be changed. It happens that into $JENKINS_HOME/jobs/ folder there is a list of folder, each for every job, and $JENKINS_HOME/jobs/*/config.xml contains the configuration, and there is something like:</p> <pre class="wp-block-code"><code> <definition class="org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition" plugin="workflow-cps@2.63"> <scm class="hudson.plugins.git.GitSCM" plugin="git@3.9.3"> <configVersion>2</configVersion> <userRemoteConfigs> <hudson.plugins.git.UserRemoteConfig> <url>ssh://git@172.17.0.2:22/xWave/orders.git</url> ...</code></pre> <p class="wp-block-paragraph">I proceeded with an update script:</p> <pre class="wp-block-preformatted">#!/bin/sh for file in `ls */config.xml`; do sed -i -e 's/ssh:\/\/git@172.17.0.2/ssh:\/\/git@gogs/g' $file done</pre> <p class="wp-block-paragraph">This was enough.</p> <p class="wp-block-paragraph"><span style="text-decoration: underline;">Now gogs</span>. It is problematic, really, but it happens that gogs uses sqlite for storing information about webhook used to signal jenkins to start a build script.<br> So I downloaded the gogs.db from $GOGSHOME/gogs/data/gogs.db and updated it with a query in sqlitebrower (desktop app):</p> <blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>UPDATE webhook SET url = replace(url, ‘172.17.0.4’, ‘jenkins’);</p></blockquote> <p class="wp-block-paragraph">And that was enough for the first day, in 4 hours the services was back and working again, but … but Jenkins refuse to work.</p> <p class="wp-block-paragraph">Additional problem with Jenkins. It was started as root, and now in swarm mode I do not want such a thing. I was using a dirty solution provided by digital ocean just for testing (<em>I am not blaming digital ocean for it, it was stated clearly!</em>), and now nothing worked.</p> <p class="wp-block-paragraph">Also I was using <code>pdmlab/jenkins-node-docker-agent:6.11.1</code> image as pipeline agent because I need to run some integration testing before setting service on other swarm machine.</p> <p class="wp-block-paragraph">First of all I found the source of digital ocean provided image, and modified that adding something</p> <p class="wp-block-paragraph">RUN apk -U add docker shadow \<br> && rm -rf /var/cache/apk/* \<br> && addgroup jenkins ping \<br> && addgroup jenkins shadow</p> <p class="wp-block-paragraph">I needed to add ping, that is 999 in <code>jenkins/jenkins:lts-alpine</code> image, but it is docker in the hosting machine. That seems to work.<br> But still there was problem using docker-compose with that image <a href="https://github.com/PDMLab/jenkins-node-docker-agent">https://github.com/PDMLab/jenkins-node-docker-agent</a></p> <p class="wp-block-paragraph">After some hours working on it I realized that that image was an overkiller, it runs a dockerd inside a container (dind – docker in a docker), but for my use I was mounting the /var/run/docker.sock inside, and I do not know if dockerd daemon just quits because it found it is already running (on host machine), or whatelse.</p> <p class="wp-block-paragraph">I fact I run jenkins container (now as service) with ‘-v /var/run/docker.sock:/var/run/docker.sock’ that is good for jenkins, and then jenkins pass the same to the docker agent.</p> <p class="wp-block-paragraph">Also I do not need nodejs, but just docker-compose. I started to write my own version, I face some problems and learnt some lessons.</p> <p class="wp-block-paragraph">At the end the docker image is in <a href="https://hub.docker.com/r/danielecr/dconedo">https://hub.docker.com/r/danielecr/dconedo</a><br> It is really simple.</p> <p class="wp-block-paragraph">Here is where I found the –group-add parameter to pass to the script:<br> <a href="https://github.com/jenkinsci/docker/issues/263">https://github.com/jenkinsci/docker/issues/263</a></p> <p class="wp-block-paragraph">Here is where I found how to do it:<br> <a href="https://stackoverflow.com/questions/44805076/setting-build-args-for-dockerfile-agent-using-a-jenkins-declarative-pipeline">https://stackoverflow.com/questions/44805076/setting-build-args-for-dockerfile-agent-using-a-jenkins-declarative-pipeline</a></p> <p class="wp-block-paragraph">So, with the help of the community(ies) I found the way to having back the building system, in a better shape.</p>

UPDATE repositories SET extra_info = regexp_replace(extra_info, '^(.*)extra_clone_url: ssh:\/\/git@(172.17.0.2:22)(.*)$', '\1extra_clone_url: ssh://git@gogs:22\3', 'g')

  <definition class="org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition" plugin="workflow-cps@2.63">
    <scm class="hudson.plugins.git.GitSCM" plugin="git@3.9.3">
      <configVersion>2</configVersion>
      <userRemoteConfigs>
        <hudson.plugins.git.UserRemoteConfig>
          <url>ssh://git@172.17.0.2:22/xWave/orders.git</url>
...

#!/bin/sh

for file in `ls */config.xml`; do
        sed -i -e 's/ssh:\/\/git@172.17.0.2/ssh:\/\/git@gogs/g' $file
done