Moving rpg source from IBM i to github

[lgiammattei]

Hellò, in my shop we are thinking about moving our rpg(le), cl(le), dds and ddl souce file from iseries souce phisical file to a public repository such github or another one.
Do you know if is there any public tutorial, guide or whatever about this topic? I tried seaching with google but i haven't found anything really usefull.
I am trying to understand how compile and moving to production environment will work.

[priosoft]

I follow your question too

[JDubbTX]

There are several ways to achieve this, and the best answer for you depends on your shop. If you already have a vendor relationship with companies like ARCAD, MDCMS, ROCKET, etc, I would check with them first, as they likely have tools for this.

You must also realize that simply copying the source to the IFS is likely not going to be enough, depending on the build tools you plan on using. Getting the source to the IFS and compiling from it is easy, coming up with a build and deploy pipeline is more difficult - which is usually where vendors come in.

If you are going it alone, here are a couple of free options to get you started:

Option 1.

@worksofliam has an option named 'migrate'. I haven't personally used this, but as I recommend pretty much anything Liam does, I'm sure it works beautifully. Look at the project readme page for instructions, seems pretty straight forward.
https://github.com/worksofliam/migrate

Liam typically uses 'make' files to build his objects. Here is a sample Github project that uses make: https://github.com/IBM/ibmi-company_system

Option 2.

If you plan on using BOB as your build tool, there is a BOB command makei cvtsrcpf. BOB builds make files under the covers, eliminating the often tedious work of coming up with makefiles.

https://ibm.github.io/ibmi-bob/#/prepare-the-project/convert-source-code

I have done this and it converted thousands of our company's source members to stream file in pretty quick time - less than 5 minutes.
Again, realize that there may be some additional renaming of objects you need to do if you want to use BOB. The cvtsrcpf option assumes that your rpgle members will become modules, thus naming them with a *.RPGLE file extension. If you want BOB to build a bound ILE program (with no module) you have to rename the file extension to *.PGM.RPGLE. All include members have to be renamed as well. Its all mentioned in the documentation.

With BOB, there is other setup needed as well, such as creating the project metadata file iproj.json, as well as the different rules.mk files. Its all outlined in the documentation.

Here is an example github repo that uses BOB as the build tool, with examples of all the different types of objects that BOB can make: https://github.com/edmundreinhardt/bob-recursive-example

[NicholasLanders]

I have personally used Option 1 with great experience thus far. I work totally out of the IFS, and use the make file to get the objects to compile the way I want to and also move the source members into the AS400. My current team does not use this, so this has allowed me to work the way that I like to (using GIT and such) while still having all my members in the AS400.

[JDubbTX]

This is awesome - many folks probably feel caged by using whatever source code management their company uses, but its cool to here about folks that can pull GIT into an existing construct and make it better. @NicholasLanders do you have anything published with a sample make file that you could share?

[NicholasLanders]

I do not, I use my corporate bitbucket account for my code. But my make file was based right off of Liam's examples. That is how I built mine.

[jwoehr]

@lgiammattei ...

• If you're going to move things over to the IFS permanently and work from IFS directories, it's pretty painless: run CPYTOSTMF on each of your members. I've written little programs from time to time to iterate over a source physical file and do this.
• If you want to use git and stay in the OFS then there is IBM Champion @richardschoen 's commercial-but-reasonable offering iForgit

Everything @JDubbTX said is good as well.

[jwoehr]

Oops, corrected, meant CPYTOSTMF not CPYFRMSTMF !

[edmundreinhardt]

As @JDubbTX said, BOB offers this support with cvttostmf. It supports a new IFS git-compatible IBM i project format that the IBM i world is moving to. It is used by IBM Merlin, ARCAD and BOB and other vendors are lining up. There is a VS Code extension that is being used by all of them https://ibm.github.io/vscode-ibmi-projectexplorer/#/
The section on migration is here https://ibm.github.io/vscode-ibmi-projectexplorer/#/pages/projectExplorer/migrate-source
Not only does it offer UI to migrate your source, it is incorporating new technology from Liam call Source Orbit that does the necessary fixing of include statements and file extensions. (The latter is still in beta status but is very exciting)

[lgiammattei]

Thank you all, so many interesting ideas and aspects to reflect on.
So, summing up to see if I understand the steps needed to start using an external source code repository.

1. Create the external source code repository and connect it to the IFS of our Power I server.
2. Migrate the source libraries (we have more than one, we manage multiple companies with multiple operating environments each) in the IFS to the folder connected to the external repository.
3. Depending on the tool used for the SRC-PF migration to IFS and the one that will subsequently be used in the future to build the objects to be executed, it is necessary to pay attention to the naming standard of the sources. In fact, we have both single-module and multi-module programs, just as we also have service programs made up of multiple modules. Oh and we use a lot of /Copy and /Include directives in our sources.
   In my shop we use a development standard that provides a different source file for each module.
   We have adopted a naming standard for the various objects to summarize their use and functionality. And here is where there may be problems that will need to be addressed and resolved depending on the type of object construction tool we choose to use.
4. Once the conversion to IFS has been completed and the object names have been adjusted if necessary, the external source repository will also be populated.
5. At this point the IDEs used by coders, mostly Rational and more recently VS Code, must point to this external source repository. Subsequent development will be local to the developers' computers who will then commit the changed or new programs.
6. The folder structure and the sources migrated to the IFS will be used for compiling the modified and written sources from scratch.
7. So, I agree with @JDubbTX the real deal is to choose the tool for the subsequent management of the builds, because each tool has its own standards and operating conventions and we will inevitably have to modify ours to adapt to the new context.
8. Since the initial post of this thread I have seen a demo of one of the products indicated by @JDubbTX and done various tests with the open source ones and ultimately everyone predicts that the time taken for this migration will be a tough and complicated time.

[andreas-prouza]

I have also published a sample project using git and make:
https://github.com/andreas-prouza/ibm-i-build

Other topics include:

• Dependencies
• Performance
• Usability
• Deployment

General

If you want to use this as a common tool in your company, you should think about the strategy of how to integrate it into your current development.
Every developer should be able to use it.

Since I use git, I work with all sources directly on my PC.
Modified sources are only synchronized to the IBM i only for compilation purposes. (This is done automatically)
With this setup, I am very fast with all my sources and can take advantage of many features that vscode comes with.

Dependencies

In your makefiles you can define dependencies of objects.
This means that if you change a PF, LF, DSPF, etc. 'make' can automatically compile all programs that depend on it.
You can generate this list using an APIs, such as DSPPGMREF, with a small program.
The question is:

• Who is generating/extending it?
• When should it be generated/extended? (What should happen if you introduce a new PF/LF or a program that depands on a PF/LF?)

Performance

Usually, you will use make on your IBM i.
If you want to use the dependency checking feature of 'make' (and you should), this can lead to performance problems if you have thousands of sources but only a small IBM i to compile them (it may take several minutes just to check the dependencies).
You can test the performance by running a 'dry' (simulated) compile. This just prints out the compile commands, but does not execute them.
I have no experience here with BOB, as this project doesn't meet all my needs.

Since almost any PC can handle this kind of IFS/PASE work better than the IBM i, I moved this work to the PC.
Now all the compilation commands are generated in the correct order on the PC (within a second) and executed on the IBM i.

Usability

You should ask yourself which way of developing (& deploying) can make your life easier and more comfortable.
git and 'make' has a lot of features (e.g. branches, dependency list, traceability, ...)

What happens if 2 developer are working on the same source and they need to be merged?
git do merges automatically when possible, but should it?
There should also a merge strategy discussed in the team.

I am using it for years and development makes much more fun since I moved to them.
There are so many possibilities, because it's all open to extend it with your own features like an extended source list (have a look at my published project at the bottom).
The extended source list I introduced, because there was no source description (as we had it in PDM) after moving to IFS.

Deployment

For this I use a separate branch.
All changes first need to get into this branch.
Then the compile commands will be generated from that branch.
Only sources where the modified timestamp is younger than the last build timestamp will be compiled.