Skip to content

feat(key on repo url): support git hosts other than GitHub + multiple forks #1043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 92 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
9bd048c
feat(key on repo url): start of db refactor (WIP)
kriswest May 19, 2025
4891468
feat(key on repo url): consolidate validation for repo creation to re…
kriswest May 19, 2025
1ad07ab
feat(key on repo url): add fallback to non-url repo matching
StingRayZA May 20, 2025
5eeccc8
feat(key on repo url): switch repo detail used in queries from name t…
kriswest May 20, 2025
a3bdb7e
feat(key on repo url): refactor tests to use repo URLs rather than na…
kriswest May 20, 2025
522ab1c
feat(key on repo url): switch repo indexing to url field and make uni…
kriswest May 20, 2025
3433157
feat(key on repo url): allow file db to delete multiple records
StingRayZA May 20, 2025
3aaa6f7
feat(key on repo url): switching to key on _id for in API + consolida…
kriswest May 20, 2025
e73fba4
feat(key on repo url): update calls in cli tests to use repo ids
StingRayZA May 21, 2025
a2f1e6c
feat(key on repo url): apply proper typing to DB classes
kriswest May 22, 2025
62938c7
feat(key on repo url): refactoring proxy to support embedding host wi…
kriswest May 28, 2025
350bb0c
feat(key on repo url): remove defunct config variable
kriswest May 29, 2025
b8973bb
feat(key on repo url): remove unneeded backwards compat in DB
kriswest May 29, 2025
b48df7a
feat(key on repo url): resolving issues in refactored proxy based on …
kriswest May 29, 2025
451b18a
feat(key on repo url): refactor UI to use repo Ids, updated urls and …
kriswest May 29, 2025
a4d9bde
feat(key on repo url): fix issues in CLI tests related to change of k…
kriswest May 30, 2025
a5beb57
fix: add indexes, compaction and consistent lowercasing of inputs in …
kriswest Mar 25, 2025
4b305dd
test: use unique emails for users in tests and remove afterwards
kriswest Apr 1, 2025
ecee528
fix: add indexes, compaction and consistent lowercasing of inputs in …
kriswest Mar 25, 2025
092435d
test: more code coverage in DB and service/routes/repo
kriswest Apr 16, 2025
a44e3d1
test: more DB test coverage
kriswest Apr 17, 2025
97115d6
feat(key on repo url): refactor tests to use repo URLs rather than na…
kriswest May 20, 2025
2e0f396
feat(key on repo url): allow file db to delete multiple records
StingRayZA May 20, 2025
ddb823e
feat(key on repo url): switching to key on _id for in API + consolida…
kriswest May 20, 2025
928823d
feat(key on repo url): apply proper typing to DB classes
kriswest May 22, 2025
4905970
feat(key on repo url): refactoring proxy to support embedding host wi…
kriswest May 28, 2025
69d83d0
feat(key on repo url): resolving issues in refactored proxy based on …
kriswest May 29, 2025
561f069
feat(key on repo url): fix CLI and cypress tests after change to prox…
kriswest Jun 3, 2025
0f51320
chore(key on repo url): clean-up unused helper function and test
kriswest Jun 3, 2025
e50c918
fix: adding URL sanitisation to resolve codeQL flagged issues
kriswest Jun 3, 2025
e318dc6
fix(key on repo url): adjust cypress test broken by changes to id's used
kriswest Jun 5, 2025
2ebdbf6
fix: resolving issues with tests after merging main + output string w…
kriswest Jun 5, 2025
f7b1911
fix(key on repo url): further tweaks to failing tests
kriswest Jun 5, 2025
46bdc9c
fix: restart the proxy when handling a new origin
kriswest Jun 11, 2025
c54e632
test: correcting git URLs in tests and increasing coverage on parseAc…
kriswest Jun 12, 2025
0976395
test: fixing issues in pasreAction and db pojo tests
kriswest Jun 12, 2025
c616f06
test: data setup for parseAction test
kriswest Jun 12, 2025
eb42b94
test: enable tests that weren't being run due to file name
kriswest Jun 12, 2025
d7d7e23
chore: prettier
kriswest Jun 16, 2025
eb22bcb
fix: ensure proxy routes are reworked on restart after adding a repo
kriswest Jun 18, 2025
4265fcb
test: improve coverage for db and repo API
kriswest Jun 18, 2025
662830c
test: proxy route filter tests
kriswest Jun 20, 2025
60a2c37
fix: correct comparison of origins and import of proxy class in repo …
kriswest Jun 24, 2025
92b801b
fix: remove Transfer-encoding from proxy blocked response
kriswest Jul 4, 2025
e5cb9c9
Merge branch 'main' into 950-key-on-repo-urls
kriswest Jul 4, 2025
fe934df
fix: log warnign on startup if proxyUrl is in the config
kriswest Jul 4, 2025
e059206
fix: catch errors on building unique DB indices
kriswest Jul 4, 2025
a462ed6
fix: control max length of urls processed to prevent DoS
kriswest Jul 4, 2025
7ab7df2
test: test coverage for path length limits
kriswest Jul 4, 2025
9a1e653
fix: typing issue in db client
kriswest Jul 4, 2025
6ad79ab
fix: fix issues with restart of the proxy by passing service a refere…
kriswest Jul 8, 2025
d07b132
fix: do proxy prep before initializing proxy to catch configured proj…
kriswest Jul 8, 2025
4828135
test: comments on proxyRoute tests
kriswest Jul 8, 2025
4a11817
test: shorten test timeouts
kriswest Jul 8, 2025
a676c22
Merge branch 'main' into 950-key-on-repo-urls
kriswest Jul 9, 2025
03bcc6e
feat: support for gitlab user profile links
kriswest Jul 10, 2025
07ab1cb
feat: add support for retrieving repo details from gitlab and fix sma…
kriswest Jul 10, 2025
d2747b9
Merge branch 'main' into 950-key-on-repo-urls
kriswest Jul 10, 2025
097f35d
Merge branch 'main' into 950-key-on-repo-urls
kriswest Jul 10, 2025
49be61f
Merge branch 'main' into 950-key-on-repo-urls
kriswest Jul 11, 2025
61412fe
feat: tweaks to types needed after merge
kriswest Jul 11, 2025
29b0a8c
Merge branch 'main' into 950-key-on-repo-urls
kriswest Jul 11, 2025
6dac8df
test: fix a test (that was retrieving a repo by name) after merging main
kriswest Jul 11, 2025
004b644
chore: move uril functions to utils.tsx and add comments
kriswest Jul 11, 2025
e677e61
feat: rework remote repository metadata handling
kriswest Jul 11, 2025
3296af6
fix: make sure user profile is available after login
kriswest Jul 14, 2025
da0a0db
Merge remote-tracking branch 'finos/main' into 950-key-on-repo-urls
kriswest Jul 14, 2025
baf50b9
fix: fix missing snackbar on login failure
kriswest Jul 14, 2025
172895c
fix: resolve RepoDetails view display issues
kriswest Jul 15, 2025
7e4cf5a
fix: typing issues in UI views
kriswest Jul 16, 2025
1034ad7
fix: more typing issues in ui and matching code button dimensions to …
kriswest Jul 16, 2025
8105796
Merge branch 'main' into 950-key-on-repo-urls
kriswest Jul 21, 2025
837b49c
Merge branch 'main' into 950-key-on-repo-urls
kriswest Jul 29, 2025
7ec3626
Merge branch 'main' into 950-key-on-repo-urls
kriswest Jul 31, 2025
4cb7a89
Merge branch 'main' into 950-key-on-repo-urls
kriswest Jul 31, 2025
88330d8
Merge branch 'main' into 950-key-on-repo-urls
kriswest Aug 5, 2025
82c6160
Merge branch 'main' into 950-key-on-repo-urls
kriswest Aug 5, 2025
63fa968
test: switch gitlab test to a smaller gitlab repo
kriswest Aug 5, 2025
47fedf4
Merge branch '950-key-on-repo-urls' of https://github.com/kriswest/gi…
kriswest Aug 5, 2025
12f7ad5
fix: clean-up of proxy filter logging
kriswest Aug 5, 2025
7e054ed
Merge branch '946-associate-commits-by-email-rebase' into natwest-dep…
kriswest Aug 5, 2025
90fa5af
Merge branch '950-key-on-repo-urls' into natwest-deployment-candidate-5
kriswest Aug 5, 2025
369d78e
test: resolve test issues after merge of fixes for 946 and 950
kriswest Aug 5, 2025
e612ad8
test: fix minor typo in a test title
kriswest Aug 5, 2025
8b33139
fix: reduce error msg from snackbar to console warn
kriswest Aug 5, 2025
e7c3e74
Merge branch '950-key-on-repo-urls' into 950-key-on-repo-urls-merge-946
kriswest Aug 5, 2025
c11637e
Merge branch 'main' into 950-key-on-repo-urls
kriswest Aug 7, 2025
7ed7d46
Merge branch 'main' into 950-key-on-repo-urls-merge-946
kriswest Aug 7, 2025
3b24240
Merge branch '950-key-on-repo-urls-merge-946' into 950-key-on-repo-urls
kriswest Aug 7, 2025
06ad5b6
fix: add examples to new repo field hints
kriswest Aug 8, 2025
804a884
fix: make trimTrailingDotGit null safe
kriswest Aug 8, 2025
256ef3f
fix: add constant and comments for max URL length in proxy helpers
kriswest Aug 8, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion config.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,11 @@
"description": "Configuration for customizing git-proxy",
"type": "object",
"properties": {
"proxyUrl": { "type": "string" },
"proxyUrl": {
"type": "string",
"description": "Used in early versions of git proxy to configure the remote host that traffic is proxied to. In later versions, the repository URL is used to determine the domain proxied, allowing multiple hosts to be proxied by one instance.",
"deprecated": true
},
"cookieSecret": { "type": "string" },
"sessionMaxAgeHours": { "type": "number" },
"api": {
Expand Down
10 changes: 4 additions & 6 deletions cypress/e2e/repo.cy.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ describe('Repo', () => {

describe('Code button for repo row', () => {
it('Opens tooltip with correct content and can copy', () => {
const cloneURL = 'http://localhost:8000/finos/git-proxy.git';
const cloneURLRegex = /http:\/\/localhost:8000\/(?:[^\/]+\/).+\.git/;
const tooltipQuery = 'div[role="tooltip"]';

cy
Expand All @@ -19,10 +19,8 @@ describe('Repo', () => {
.should('not.exist');

cy
// find the entry for finos/git-proxy
.get('a[href="/dashboard/repo/git-proxy"]')
// take it's parent row
.closest('tr')
// find a table row for a repo (any will do)
.get('table#RepoListTable>tbody>tr')
// find the nearby span containing Code we can click to open the tooltip
.find('span')
.contains('Code')
Expand All @@ -35,7 +33,7 @@ describe('Repo', () => {
.should('exist')
.find('span')
// check it contains the url we expect
.contains(cloneURL)
.contains(cloneURLRegex)
.should('exist')
.parent()
// find the adjacent span that contains the svg
Expand Down
7 changes: 4 additions & 3 deletions index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import { hideBin } from 'yargs/helpers';
import * as fs from 'fs';
import { configFile, setConfigFile, validate } from './src/config/file';
import { initUserConfig } from './src/config';
import proxy from './src/proxy';
import Proxy from './src/proxy';
import service from './src/service';

const argv = yargs(hideBin(process.argv))
Expand All @@ -30,7 +30,7 @@ const argv = yargs(hideBin(process.argv))
.strict()
.parseSync();

setConfigFile(argv.c as string || "");
setConfigFile((argv.c as string) || '');
initUserConfig();

if (argv.v) {
Expand All @@ -48,7 +48,8 @@ if (argv.v) {

validate();

const proxy = new Proxy();
proxy.start();
service.start();
service.start(proxy);

export { proxy, service };
20 changes: 7 additions & 13 deletions packages/git-proxy-cli/test/testCli.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -221,13 +221,13 @@ describe('test git-proxy-cli', function () {
before(async function () {
await helper.addRepoToDb(TEST_REPO_CONFIG);
await helper.addUserToDb(TEST_USER, TEST_PASSWORD, TEST_EMAIL, TEST_GIT_ACCOUNT);
await helper.addGitPushToDb(pushId, TEST_USER, TEST_EMAIL, TEST_REPO);
await helper.addGitPushToDb(pushId, TEST_REPO_CONFIG.url, TEST_USER, TEST_EMAIL);
});

after(async function () {
await helper.removeGitPushFromDb(pushId);
await helper.removeUserFromDb(TEST_USER);
await helper.removeRepoFromDb(TEST_REPO_CONFIG.name);
await helper.removeRepoFromDb(TEST_REPO_CONFIG.url);
});

it('attempt to authorise should fail when server is down', async function () {
Expand Down Expand Up @@ -304,7 +304,7 @@ describe('test git-proxy-cli', function () {
after(async function () {
await helper.removeGitPushFromDb(pushId);
await helper.removeUserFromDb(TEST_USER);
await helper.removeRepoFromDb(TEST_REPO_CONFIG.name);
await helper.removeRepoFromDb(TEST_REPO_CONFIG.url);
});

it('attempt to cancel should fail when server is down', async function () {
Expand Down Expand Up @@ -421,13 +421,13 @@ describe('test git-proxy-cli', function () {
before(async function () {
await helper.addRepoToDb(TEST_REPO_CONFIG);
await helper.addUserToDb(TEST_USER, TEST_PASSWORD, TEST_EMAIL, TEST_GIT_ACCOUNT);
await helper.addGitPushToDb(pushId, TEST_USER, TEST_EMAIL, TEST_REPO);
await helper.addGitPushToDb(pushId, TEST_REPO_CONFIG.url, TEST_USER, TEST_EMAIL);
});

after(async function () {
await helper.removeGitPushFromDb(pushId);
await helper.removeUserFromDb(TEST_USER);
await helper.removeRepoFromDb(TEST_REPO_CONFIG.name);
await helper.removeRepoFromDb(TEST_REPO_CONFIG.url);
});

it('attempt to reject should fail when server is down', async function () {
Expand Down Expand Up @@ -498,19 +498,13 @@ describe('test git-proxy-cli', function () {
before(async function () {
await helper.addRepoToDb(TEST_REPO_CONFIG);
await helper.addUserToDb(TEST_USER, TEST_PASSWORD, TEST_EMAIL, TEST_GIT_ACCOUNT);
await helper.addGitPushToDb(pushId, TEST_REPO, TEST_USER, TEST_EMAIL);
await helper.addGitPushToDb(pushId, TEST_REPO_CONFIG.url, TEST_USER, TEST_EMAIL);
});

after(async function () {
await helper.removeGitPushFromDb(pushId);
await helper.removeUserFromDb(TEST_USER);
await helper.removeRepoFromDb(TEST_REPO_CONFIG.name);
});

after(async function () {
await helper.removeUserFromDb('testuser1');
await helper.removeGitPushFromDb(pushId);
await helper.removeRepoFromDb(TEST_REPO_CONFIG.name);
await helper.removeRepoFromDb(TEST_REPO_CONFIG.url);
});

it('attempt to ls should list existing push', async function () {
Expand Down
20 changes: 11 additions & 9 deletions packages/git-proxy-cli/test/testCliUtils.js
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ async function runCli(
expectedExitCode = 0,
expectedMessages = null,
expectedErrorMessages = null,
debug = false,
debug = true,
) {
try {
console.log(`cli: '${cli}'`);
Expand Down Expand Up @@ -152,8 +152,9 @@ async function addRepoToDb(newRepo, debug = false) {
const found = repos.find((y) => y.project === newRepo.project && newRepo.name === y.name);
if (!found) {
await db.createRepo(newRepo);
await db.addUserCanPush(newRepo.name, 'admin');
await db.addUserCanAuthorise(newRepo.name, 'admin');
const repo = await db.getRepoByUrl(newRepo.url);
await db.addUserCanPush(repo._id, 'admin');
await db.addUserCanAuthorise(repo._id, 'admin');
if (debug) {
console.log(`New repo added to database: ${newRepo}`);
}
Expand All @@ -166,27 +167,28 @@ async function addRepoToDb(newRepo, debug = false) {

/**
* Removes a repo from the DB.
* @param {string} repoName The name of the repo to remove.
* @param {string} repoUrl The url of the repo to remove.
*/
async function removeRepoFromDb(repoName) {
await db.deleteRepo(repoName);
async function removeRepoFromDb(repoUrl) {
const repo = await db.getRepoByUrl(repoUrl);
await db.deleteRepo(repo._id);
}

/**
* Add a new git push record to the database.
* @param {string} id The ID of the git push.
* @param {string} repo The repository of the git push.
* @param {string} repoUrl The repository URL of the git push.
* @param {string} user The user who pushed the git push.
* @param {string} userEmail The email of the user who pushed the git push.
* @param {boolean} debug Flag to enable logging for debugging.
*/
async function addGitPushToDb(id, repo, user = null, userEmail = null, debug = false) {
async function addGitPushToDb(id, repoUrl, user = null, userEmail = null, debug = false) {
const action = new actions.Action(
id,
'push', // type
'get', // method
Date.now(), // timestamp
repo,
repoUrl,
);
action.user = user;
action.userEmail = userEmail;
Expand Down
1 change: 0 additions & 1 deletion proxy.config.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
{
"proxyUrl": "https://github.com",
"cookieSecret": "cookie secret",
"sessionMaxAgeHours": 12,
"rateLimit": {
Expand Down
10 changes: 0 additions & 10 deletions src/config/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ let _database: Database[] = defaultSettings.sink;
let _authentication: Authentication[] = defaultSettings.authentication;
let _apiAuthentication: Authentication[] = defaultSettings.apiAuthentication;
let _tempPassword: TempPasswordConfig = defaultSettings.tempPassword;
let _proxyUrl = defaultSettings.proxyUrl;
let _api: Record<string, unknown> = defaultSettings.api;
let _cookieSecret: string = serverConfig.GIT_PROXY_COOKIE_SECRET || defaultSettings.cookieSecret;
let _sessionMaxAgeHours: number = defaultSettings.sessionMaxAgeHours;
Expand All @@ -54,15 +53,6 @@ let _config = { ...defaultSettings, ...(_userSettings || {}) } as Configuration;
// Create config loader instance
const configLoader = new ConfigLoader(_config);

// Get configured proxy URL
export const getProxyUrl = () => {
if (_userSettings !== null && _userSettings.proxyUrl) {
_proxyUrl = _userSettings.proxyUrl;
}

return _proxyUrl;
};

// Gets a list of authorised repositories
export const getAuthorisedList = () => {
if (_userSettings !== null && _userSettings.authorisedList) {
Expand Down
26 changes: 12 additions & 14 deletions src/db/file/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,27 @@ import * as users from './users';
import * as repo from './repo';
import * as pushes from './pushes';

export const {
getPushes,
writeAudit,
getPush,
deletePush,
authorise,
cancel,
reject,
canUserCancelPush,
canUserApproveRejectPush,
} = pushes;
export const { getPushes, writeAudit, getPush, deletePush, authorise, cancel, reject } = pushes;

export const {
getRepos,
getRepo,
getRepoByUrl,
getRepoById,
createRepo,
addUserCanPush,
addUserCanAuthorise,
removeUserCanPush,
removeUserCanAuthorise,
deleteRepo,
isUserPushAllowed,
canUserApproveRejectPushRepo,
} = repo;

export const { findUser, findUserByOIDC, getUsers, createUser, deleteUser, updateUser } = users;
export const {
findUser,
findUserByEmail,
findUserByOIDC,
getUsers,
createUser,
deleteUser,
updateUser,
} = users;
Loading
Loading