Automation: Push generated Code into Space git Repo

Hi - is there any update on this question,  I would similarily like the job to be able to commit back a file (and tags) to the git repository Im working with? Should I generate an access token and try and use that to auth back?

Philip - hi,

I apologize for the late response.

The feature to push into the repository from a job is planned. One of the major parts that is required for it is in progress right now but there's no ETA for the full feature, unfortunately.

As for the access token - if there is a need to push into a repository, credentials should be provided using Secrets. Additionally, please make sure that there's a git client installed in the container that is used.

Margarita Kolotilova
JetBrains
http://www.jetbrains.com
The Drive to Develop

Hi Margarita Kolotilova is there any update for pushing into repositories?   

Margarita Kolotilova : Is there any update on this. When we build source code using pipeline we want to tag the commit on successful build or if we have golang code on successful build in pipeline we need to tag the commit  so that other projects can fetch that gomodule by tag which is not happening because this feature is not available to push tags to the current repo. 

Chaitu the workaround that we have to do was cloning the repository again in the same pipeline and pushing inside the new repository folder. 

Hi all,

Any new update on the progress of this task? Margarita Kolotilova

Hello everyone!

As for pushing to a repository from a job, the general approach is as following:

1. In case you'd like to push to an external repository: it's necessary to provide a private key as a Secret into the job, it might look something like this:

job("Work with external repo") {
    container("alpine/git") {
        env["REPO_PRIVATE_KEY"] = Secrets("repo-private-key-ssh")

        shellScript {
            content = """
                mkdir -p ~/.ssh
                echo ${'$'}APP_PRIVATE_KEY >> ~/.ssh/id_rsa
                chmod 400 ~/.ssh/id_rsa
                ssh-keyscan -t rsa git.my-org.com >> ~/.ssh/known_hosts
                
                cd ~
                git clone ssh://git@git.my-org.com/my-repo.git
                
                cd my-repo
                git commit ...
                git push ...
            """
        }
    }
}

2. In case you'd like to push to the repository for which the job is run or to another repository that is also hosted in Space, the idea is basically the same, but as an additional step user should create an application https://www.jetbrains.com/help/space/applications.html#get-started-with-applications, give it rights to the required repositories and add a public key to its ssh keys. The only caveat here is that if it's necessary to push to the same repository from which the job is run and there is a generic gitPush trigger there it can enter an endless loop. We don't have out of the box solution for this, so it's necessary to introduce a condition to restrict from running a job once again after committing into the repository from a pervious job.

If you have any specific questions/issues with the approach described above please file a separate support ticket via https://jb.gg/space-support and we'll be happy to analyze your exact case and offer a solution.

Margarita Kolotilova
JetBrains
http://www.jetbrains.com
The Drive to Develop

Anyone else getting the error "Load key "/root/.ssh/id_rsa": invalid format" using this approach and know how to solve it?

and for anyone else trying this, make sure to use the correct env variable in both places. REPO_PRIVATE_KEY / APP_PRIVATE_KEY

Stefan - hi,

Could you please create a support request via this form https://jb.gg/space-support providing a bit more details on the issue you have? So we can take a closer look at the problem?

Margarita Kolotilova
JetBrains
http://www.jetbrains.com
The Drive to Develop

Hi Stefan!

It's maybe too late for you, but I think I found a solution. When you paste your private key file content into a secret, the new line characters will dissapear. You can check it as follows:

...
echo ${'$'}APP_PRIVATE_KEY >> ~/.ssh/id_rsa
tail ~/.ssh/id_rsa
...

My solution was to create a custom container and copy the key file as I created it (I was already using a custom container). My Dockerfile looks like this:

FROM alpine/git
RUN mkdir -p /root/.ssh
COPY ./host/machine/key/path/id_rsa /root/.ssh/id_rsa
RUN chmod 400 /root/.ssh/id_rsa
RUN ssh-keyscan -t rsa git.myorg.com >> /root/.ssh/known_hosts

And my job script:

job("Work with external repo") {
    container("custom/container") {
        shellScript {
            content = """
                cd /root
                git clone ssh://git@git.my-org.com/my-repo.git
                
                cd my-repo
                git status
            """
        }
    }
}

Hope it helps,

Attila

Thank you @Almaxbacsi. I ended up base64 encoding the secret and then decoding it, but maybe i should add it as a normal string again and then append a new line at the end of it just to keep it as a secret so it wont leak out to any logs.

Margarita Kolotilova you should not trim secrets entered by users :) 

For anyone else that lands here, between the above and a whole lot of trial and error I got this working for a Space repository:

• Generate an SSH Key Pair, add the public key to the Application (referenced by Margarita above) 
• Base64 encode your private key and add it as a secret in your project. This part is important, as formatting is very important.

Then your shell script:

env["REPO_PRIVATE_KEY"] = Secrets("repo-private-key-base64")

shellScript{

SSH_PRIVATE_KEY=$(echo ${'$'}REPO_PRIVATE_KEY | base64 -d)

mkdir -p ~/.ssh

echo "${'$'}SSH_PRIVATE_KEY" > ~/.ssh/id_rsa

chmod 600 ~/.ssh/id_rsa

ssh-keyscan -t rsa git.jetbrains.space >> ~/.ssh/known_hosts

cd ~

git config --global user.email “USER IN APPLICATION”

git clone <repo>

cd <repo>

//do stuff in repo

}

Also how do you markdown as code? I tried ``` but no luck.

I cannot get SSH to work (neither base64 nor raw). 

Another way worked: Generating a permanent token in the Application and setting the same token in project→settings→secrets. My code looks like this:

container("alpine/git") {

  env["PERMANENT_TOKEN"] = "{{ project:permanent-token-for-automation-git-application }}"
  shellScript {
  
        content = """
            cd ~
            git config --global user.email “gitaccessforautomation@my-organization.com”        
            PERM_TOKEN=${'$'}(echo ${'$'}PERMANENT_TOKEN)
            git clone  https://none:${'$'}PERM_TOKEN@git.jetbrains.space/myorganization/project/reponame.git
            tag=${'$'}(tail -n 1 /mnt/space/work/reponame/release-versions.yaml)
            git tag -a ${"$"}tag -m "tagging test with tag: ${"$"}tag"
            git push origin ${"$"}tag

            cd reponame
            cat > foo.txt
            echo "Some content " > foo.txt
            git checkout -b test
            git add foo.txt
            git commit -m "test commit"
            git push origin test
            
        """
    }
}
 

Pushing and tagging work. The commiter is “root <gitaccessforautomation@my-organization.com>”. 

Why can the automation service push to the repo when it has no permission to write to Git? (set in admin→roles→Automation Service)

Any help with understanding this or getting SSH to work is appreciated!